Red Hat Enterprise Linux Implementation

DNS Management for Servers

Module Topics

Domain Name System
Anatomy of DNS Lookups
DNS Resource Records
Caching Nameservers and DNSSEC
Configuring unbound
Dumping and Loading unbound Cache
Flushing unbound Cache
Updating Local Caching unbound Configuration With dnssec-trigger
Troubleshooting DNS
Name Resolution Methods
Client/Server Network Connectivity
DNS Response Codes
Other Common DNS Issues

Domain Name System

Domain Name System (DNS): Hierarchical naming system
- Serves as directory of networked hosts and resources
Information in directory:
- Maps network names to data
- Maintained in logical entries called resource records
DNS hierarchy:
- Begins with root domain . at top
- Branches downward to next-level domains
Each level of DNS hierarchy delineated by .
- Top level: .
- Second level: com, net, org
- Third level: example.com, redhat.com

Domain Name System

Domain

Collection of resource records that:
- Ends in common name
- Represents entire subtree of DNS name space: example.com
Largest possible domain is root domain, .
- Includes entire DNS namespace
Top-level domain (TLD) has only one component
- Generic TLDs (gTLDs) organized by theme
  - Include .com, .edu, .net, etc.
- Country code TLDs (ccTLDs) organized on national basis
  - Include .us, .uk, .cn, .ru, etc.

Domain Name System

Subdomain

Domain that is subtree of another domain
Use when discussing relationship of two domains to each other
- Example: lab.example.com is subdomain of example.com

Zone

Portion of domain for which nameserver is directly responsible (authoritative)
May be entire domain or part of domain with some or all subdomains delegated to other nameserver(s)

Anatomy of DNS Lookups

When system needs to perform name resolution using DNS server:
- Sends queries to servers listed in /etc/resolv.conf in order
- Sends until it gets response or runs out of servers
To manually look up DNS name, use host or dig

Local Authoritative Data

When query arrives at DNS server:
- Server determines whether queried information resides in zone it is authoritative for
- If yes, server responds to client with information contained in local zone file
- Server providing response is authoritative for data provided, called authoritative answer (aa)
- Authoritative answers from nameserver have aa flag in header of DNS response

Transcript:

When a system needs to perform name resolution using a DNS server, it begins by sending queries to the servers listed in /etc/resolv.conf in order, until it gets a response or runs out of servers. The host or dig commands can be used to manually look up DNS names.

When the query arrives at a DNS server, the server first determines whether the information being queried resides in a zone that it is authoritative for. If the server is an authority for the zone that the name or address being queried belongs to, then the server responds to the client with the information contained in its local zone file. This type of response is referred to as an authoritative answer (aa), since the server providing the response is authoritative for the data provided. Authoritative answers from a nameserver have the aa flag turned on in the header of the DNS response.

Anatomy of DNS Lookups

Local Cached Non-Authoritative Data

If DNS server is not authority for record but obtained record to answer previous query:
- May have copy of record in cache
  - Cache is where answers to queries stored for time specified in resource record response value Time To Live (TTL)
If answer exists in cache, server provides it to client
- Answer does not have aa flag because server is not authoritative for data being provided

Anatomy of DNS Lookups

Remote Non-Authoritative Data via Recursion

If DNS server not authoritative for name being queried and does not possess record in cache:
- Attempts to retrieve record via recursion
- DNS server with empty cache begins querying one root nameserver by IP address retrieved from local, pre-populated root hints file
- Root nameserver responds with referral to nameservers authoritative for TLD containing queried name
After receiving referral:
- DNS server performs query to TLD authoritative nameserver
- Authoritative nameserver sends either authoritative answer or another referral
- Process continues until authoritative server is reached and responds with authoritative answer
DNS server caches final answer and any intermediate answers to improve performance
- Example: During www.example.com lookup:
  - DNS server finds example.com zone has authoritative nameservers
  - DNS server queries those servers directly for future queries in example.com zone

Transcript:

If the DNS server is not authoritative for the name being queried, and it does not possess the record in its cache, it will then attempt to retrieve the record via an iterative process known as recursion. A DNS server with an empty cache begins the recursion process by querying one of the root nameservers by IP address retrieved from its local, pre-populated root hints file. The root nameserver will then likely respond with a referral, which indicates the nameservers that are authoritative for the TLD that contains the name being queried.

Upon receiving the referral, the DNS server will then perform another iterative query to the TLD authoritative nameserver it was referred to. Depending on whether there are further remaining delegations in the name being queried, this authoritative nameserver will either send an authoritative answer or yet another referral. This continues until an authoritative server is reached and responds with an authoritative answer.

The final answer, along with all the intermediate answers obtained prior to it, are cached by the DNS server to improve performance. If during a lookup for www.example.com the DNS server finds out that the example.com zone has authoritative nameservers, it will query those servers directly for any future queries for information in the example.com zone, rather than starting recursion again at the root nameservers.

DNS Resource Records

DNS resource records (RRs): Entries in DNS zone that specify information about particular name or object
Contains type, TTL, class, and data elements

RR uses this format:

owner-name           TTL      class      type     data
www.example.com.     300      IN         A        192.168.1.10

Field Name

Content

owner-name

Name for resource record

TTL

Time To Live of resource record in seconds—specifies how long DNS resolves should cache resource record

class

"Class" of record, almost always IN ("Internet")

type

Sort of information stored by this record—for example, A record maps host name to aIPv4 address

data

Data stored by record; exact format varies by record type

DNS Resource Records

A (IPv4 Address) Record

Maps host name to IPv4 address

[student@server1 ~]$ host -v -t A example.com
Trying "example.com"
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 22681
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 2

;; QUESTION SECTION:
;example.com.           IN  A

;; ANSWER SECTION:
example.com.        86400   IN  A   192.168.0.254

Recieved 96 bytes from 192.168.0.254#53 in 1 ms

DNS Resource Records

AAAA (IPv6 Address or "Quad-A") Record

Maps host name to IPv6 address

[student@server1 ~]$ host -v -t AAAA a.root-servers.net
Trying "a.root-servers.net"
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 18194
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 13, ADDITIONAL: 12

;; QUESTION SECTION:
;a.root-servers.net.        IN  AAAA

;; ANSWER SECTION:
a.root-servers.net. 604800  IN  AAAA    2001:503:ba3e::2:30

Received 64 bytes from 192.168.0.254#53 in 78 ms

DNS Resource Records

CNAME (Canoncial Name) Record

Aliases one name to another name (canonical name)
- Should have A or AAAA records
DNS resolver receives CNAME record as query response
- Reissues query using canonical name instead of original name

Data field of CNAME records can point to name anywhere in DNS

Internal or external to zone:

www-dev.example.com. IN CNAME lab.example.com.
www.example.com.     IN CNAME www.redhat.com.

DNS Resource Records

Use CNAME records with care.
Avoid pointing CNAME record to other CNAME records
Chain of CNAME record must end in A and/or AAAA records
Legitimate uses for CNAME chains:
- When using Content Delivery Networks (CDNs) to improve speed and reliability of data delivery over Internet

Point NS and MX records at names with A and/or AAAA records

Do not point NS and MX at CNAME records

[student@server1 ~]$ host -v -t A ipa-ca.server0.example.com
Trying "ipa-ca.server0.example.com"
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 11931
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 1, ADDITIONAL: 2

;; QUESTION SECTION:
;ipa-ca.server0.example.com.    IN  A

;; ANSWER SECTION:
ipa-ca.server0.example.com. 86400 IN    CNAME   server0.example.com.
server0.example.com.    86400   IN  A   192.168.0.101

Recieved 125 bytes from 192.168.0.254#53 in 1 ms

DNS Resource Records

PTR (Pointer) Record

Maps IPv4 or IPv6 addresses to host name
- Used for reverse DNS resolution
PTR records code IP address in special format that acts like host name
- IPv4 addresses: Address reversed, result treated as host in subdomain of domain in-addr.arpa
- IPv6 addresses: Address split into subdomains on nibble boundaries, set up as subdomain of domain ip6.arpa
Syntax enables DNS administrators to delegate responsibility for ranges of addresses to other DNS administrators

DNS Resource Records

Example: PTR with IPv4 address

[student@server1 ~]$ host -v -t PTR 192.168.0.1
Trying "1.0.168.192.in-addr.arpa"
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 36389
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 2

;; QUESTION SECTION:
;1.0.168.192.in-addr.arpa.  IN  PTR

;; ANSWER SECTION:
1.0.168.192.in-addr.arpa. 86400 IN  PTR desktop0.example.com.

Received 127 bytes from 192.168.0.254#53 in 2 ms

Example: PTR with IPv6 address

[student@server1 ~]$ host -v -t PTR 2001:503:ba3e::2:30
Trying "0.3.0.0.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.e.3.a.b.3.0.5.0.1.0.0.2.ip6.arpa"
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 32138
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;0.3.0.0.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.e.3.a.b.3.0.5.0.1.0.0.2.ip6.arpa. IN PTR

;; ANSWER SECTION:
0.3.0.0.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.e.3.a.b.3.0.5.0.1.0.0.2.ip6.arpa. 86400 IN PTR a.root-servers.net.

Received 122 bytes from 192.168.0.254#53 in 174 ms

DNS Resource Records

NS (Name Server) Record

Maps domain name to DNS name server authoritative for DNS zone

Every public authoritative name server for zone must have NS record

[student@server1 ~]$ host -v -t NS example.com
Trying "example.com"
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 29362
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 2

;; QUESTION SECTION:
;example.com.           IN  NS

;; ANSWER SECTION:
example.com.        86400   IN  NS  classroom.example.com.

Received 80 bytes from 192.168.0.254#53 in 0 ms

DNS Resource Records

SOA (Start of Authority) Record

Provides information about how DNS zone works
One SOA record for zone
Specifies:
- Which of zone’s name servers is primary (master)
- Information on how secondary (slave) name servers should update their information
- Zone’s management contact

DNS Resource Records

Data Element

Content

Master nameserver

Host name of nameserver that is original source of domain information and can accept dynamic DNS updates if zone supports them

RNAME

Email address of person responsible for DNS zone (hostmaster)
The @ in email address is replaced with a . in RNAME
- Example: Email address hostmaster@example.com written as hostmaster.example.com

Serial number

Version number of zone, increased when there change occurs to zone records

Refresh

How frequently slave servers should check for zone updates, in seconds

Retry

How long slave server should wait before retrying failed refresh attempt, in seconds

Expiry

If refreshes have been failing, how long slave server should wait before it stops using its old copy of zone to respond to queries, in seconds

Minimum

If resolver looks up name that does not exist (gets a nonexistent domain (NXDOMAIN) response), how long it should cache information that record does not exist, in seconds

DNS Resource Records

Example: SOA syntax

[student@server1 ~]$ host -v -t SOA example.com
Trying "example.com"
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 58434
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;example.com.           IN  SOA

;; ANSWER SECTION:
example.com.        86400   IN  SOA classroom.example.com. root.classroom.example.com. 2013091600 3600 300 604800 60

Received 121 bytes from 192.168.0.254#53 in 0 ms

DNS Resource Records

MX (Mail Exchange) Record

Maps domain name to mail exchange that accepts email for name

Data includes:

Preference number (lowest preferred) to determine order in which to pick between multiple MX records

Host name for mail exchange for that name

[student@server1 ~]$ host -v -t MX example.com
Trying "example.com"
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 47187
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 2

;; QUESTION SECTION:
;example.com.           IN  MX

;; ANSWER SECTION:
example.com.        86400   IN  MX  10 classroom.example.com.

Received 96 bytes from 192.168.0.254#53 in 0 ms

DNS Resource Records

TXT (Text) Record

Maps name to arbitrary human-readable text

Commonly used to supply data used by:

Sender Policy Framework (SPF)
DomainKeys Identified Mail (DKIM)
Domain-based Message Authentication, Reporting, and Conformance (DMARC)

Etc.

[student@server1 ~]$ host -v -t TXT lwn.net
Trying "lwn.net"
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 41137
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;lwn.net.           IN  TXT

;; ANSWER SECTION:
lwn.net.        28619   IN  TXT "v=spf1 ip4:72.51.34.34 ip4:70.33.254.29 -all"

Received 638 bytes from 192.168.2.11#53 in 74 ms

DNS Resource Records

SRV (Service) Record

Used to locate hosts that support service for domain

Using _service.protocol.domainname_, SRV records:

Provide names of hosts that provide service for domain
Provide port number that service listens on

Include priority and weight values to indicate order to use hosts when multiple hosts are available for service

[student@server1 ~]$ host -v -t server1.example.com _ldap._tcp.server0.example.com
Trying "_ldap._tcp.server0.example.com"
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 35665
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 3

;; QUESTION SECTION:
;_ldap._tcp.server0.example.com.    IN  server1.example.com

;; ANSWER SECTION:
_ldap._tcp.server0.example.com. 86400 IN server1.example.com    0 100 389 server0.example.com.

Received 154 bytes from 192.168.0.254#53 in 0 ms

DNS Resource Records

Hosts and Resource Records

Typical host has following records:
- One or more A and/or AAAA records
  - Maps host name to IP addresses
- PTR record for each IP address
  - Reverse maps them to host name
- Optionally, one or more CNAME records
  - Maps alternate names to canonical host name

DNS Resource Records

Typical DNS zone has following records (in addition to records for hosts in zone):
- One SOA record
  - Specifies how zone works
- One NS record for each authoritative name server
- One or more MX records
  - Maps domain name to mail exchange that receives email for addresses ending in domain name
- Optionally, one or more TXT records
  - For functions such as SPF or Google Site Verification
- Optionally, one or more SRV records
  - Locates services in domain

References

Man pages: host(1) and dig(1)
RFC 1034: Domain names - concepts and facilities
RFC 1035: Domain names - implementation and specification
RFC 2181: Clarifications to the DNS Specification

Caching Nameservers and DNSSEC

Caching Nameservers

Store DNS query results in local cache
Remove resource records from cache when TTLs expire
Can set up caching nameservers to perform queries on behalf of clients on local network
- Improves efficiency of DNS name resolutions
  - Reduces DNS traffic across Internet
- As cache grows, DNS performance improves
  - Nameserver answers more client queries from local cache

Caching Nameservers and DNSSEC

DNSSEC Validation

DNS transactions prone to spoofing and tampering
Caching nameservers targets of attackers looking to redirect/hijack network traffic
- Often achieve this by fooling DNS server into populating malicious data into cache: cache poisoning
- Lets attacker compromise DNS data received by clients utilizing caching name service on DNS server and then redirect/hijack clients' network traffic
Caching nameservers perform Domain Name System Security Extensions (DNSSEC) validation to improve security
- Lets caching nameserver validate authenticity/integrity of resource records before placing them in cache
- Protects clients against consequences of cache poisoning

Transcript:

Given the stateless nature of UDP, DNS transactions are prone to spoofing and tampering. Caching nameservers have historically been favored targets of attackers looking to redirect or hijack network traffic. This is often achieved by exploiting vulnerabilities in DNS server software to fool a DNS server into accepting and populating malicious data into its cache, a technique commonly referred to as cache poisoning. Once the attacker succeeds in poisoning a DNS server’s cache, they effectively compromise the DNS data received by the numerous clients utilizing the caching name service on the DNS server and can consequently redirect or hijack the clients' network traffic.

While a caching nameserver can greatly improve DNS performance on the local network, they can also provide improved security by performing Domain Name System Security Extensions (DNSSEC) validation. DNSSEC validation enabled at the caching nameserver allows the authenticity and integrity of resource records to be validated prior to being placed in the cache for use by clients, and therefore protects clients against the consequences of cache poisoning.

Configuring `unbound`

As root, install unbound:

[root@server1 ~]# yum install -y unbound

Start and enable unbound.service:

[root@server1 ~]# systemctl start unbound.service
[root@server1 ~]# systemctl enable unbound.service
ln -s '/usr/lib/systemd/system/unbound.service' '/etc/systemd/system/multi-user.target.wants/unbound.service'

Configure network interface to listen on:
- unbound listens on localhost only by default
- To make unbound available to remote clients as caching nameserver, use interface
  - Specify network interface(s) to listen on
  - 0.0.0.0 configures unbound to listen on all network interfaces
    interface: 0.0.0.0

Configuring `unbound`

Configure client access:
- unbound refuses recursive queries from all clients by default
- To specify clients allowed to make recursive queries, use access-control
  access-control: 192.168.0.0/24 allow
Configure forwarding:
- To specify DNS server(s) to forward queries to, create forward-zone clause
- To specify DNS servers by:
  - Host name, use forward-host
  - IP address, use forward-addr
- To forward all queries, use . for forward-zone
  forward-zone: name: "." forward-addr: 192.168.0.254

Configuring `unbound`

Bypass DNSSEC validation for select unsigned zones:
- unbound performs DNSSEC validation to verify all DNS responses received by default
  - To specify domain for which DNSSEC validation should be skipped, use domain-insecure
  - Often desirable with unsigned internal domain that would fail trust chain validation
    domain-insecure: example.com

Install trust anchors for signed zones without complete chain of trust:

Not all ccTLDs have completed implementation of DNSSEC
- Subdomains of these ccTLDs can be DNSSEC-signed but have broken chain of trust
To specify trust anchor for zone, use trust-anchor

To obtain DNSKEY record for key signing key (KSK) of zone, use dig and input it as trust-anchor value

[student@server1 ~]$ dig +dnssec DNSKEY example.com

trust-anchor: "example.com. 3600 IN DNSKEY 257 3 8 AwEAAawt7HplI5M8GGAsxuyCyjF0l+QlcgVN11CRZ4vP66qbDCX0BnSh Z11BGb//4zSG/8mmBHirL2FLg+mVuIIxig+iroZYjh4iTKVOhv2hZftR wyrQHK++qXvCCWN3ki51RG/e8R4kOEV71rZ8OgQvPWx6F91qroqOPpcf 7PPxippeHOn+PxnP0hpyLyo1mx1rPs/cMpL3jOMufGP+LJYh+fBU7lt0 sP5i09HaJPruzyZML9BPtpv8ZAdQhwtXVG0+MnET2qT/1+TljpxZn6ye egFRCFRHBjMo6iiRJnUWra/klkrgEn2Q+BXGTOMTTKQdYz4OxYEa1z7a pu3a09dYNBM="

Transcript:

If desired, bypass DNSSEC validation for select unsigned zones.
By default, unbound is enabled to perform DNSSEC validation to verify all DNS responses received. The domain-insecure option in the server clause of /etc/unbound/unbound.conf can be used to specify a domain for which DNSSEC validation should be skipped. This is often desirable when dealing with an unsigned internal domain that would otherwise fail trust chain validation.
If desired, install trust anchors for select signed zones without complete chain of trust.
Since not all ccTLDs have completed implementation of DNSSEC, the subdomains of these ccTLDs can be DNSSEC-signed but still have a broken chain of trust. This problem can be overcome by using the trust-anchor option in the server clause of /etc/unbound/unbound.conf to specify a trust anchor for the zone. Obtain the DNSKEY record for the key signing key (KSK) of the zone using dig and input it as the value for the trust-anchor option.

Configuring `unbound`

Save changes to /etc/unbound/unbound.conf

Check /etc/unbound/unbound.conf for syntax errors:

[root@server1 ~]# unbound-checkconf
unbound-checkconf: no errors in /etc/unbound/unbound.conf

Restart unbound.service:

[root@server1 ~]# systemctl restart unbound.service

Configure firewall to allow DNS traffic:

[root@server1 ~]# firewall-cmd --permanent --add-service=dns
success
[root@server1 ~]# firewall-cmd --reload
success

Dumping and Loading `unbound` Cache

Need to dump cache data when troubleshooting DNS issues

To dump cache with unbound, use unbound-control with dump_cache:

[root@server1 ~]# unbound-control dump_cache
START_RRSET_CACHE
;rrset 86395 1 0 3 3
classroom.example.com.  86395   IN  A   192.168.0.254
;rrset 86395 1 0 7 3
example.com.    86395   IN  NS  classroom.example.com.
;rrset 86395 1 0 8 3
example.com.    86395   IN  A   192.168.0.254
END_RRSET_CACHE
START_MSG_CACHE
msg example.com. IN A 33152 1 86395 3 1 1 1
example.com. IN A 0
example.com. IN NS 0
classroom.example.com. IN A 0
END_MSG_CACHE
EOF

This dumps cache to stdout in text format
Can direct output to file for storage
To read from stdin and load back into cache, use unbound-control load_cache:
```
[root@server1 ~]# unbound-control load_cache < dump.out
ok
```

Flushing `unbound` Cache

Periodically purge outdated resource records from cache
- Erroneous/outdated resource records prevent newly corrected counterparts from becoming available to clients until TTLs expire
To purge outdated records from cache, use unbound-control with flush:
```
[root@server1 ~]# unbound-control flush www.example.com
ok
```

To purge all resource records belonging to domain from cache, use unbound-control with flush_zone:

[root@server1 ~]# unbound-control flush_zone example.com
ok removed 3 rrsets, 1 messages and 0 key entries

Updating Local Caching `unbound` Configuration With `dnssec-trigger`

Can use unbound as local caching nameserver
- Provides secure DNS name resolution for local use on individual system
For local caching nameserver setup, point nameserver to localhost where unbound is listening
unbound configuration forwards DNS requests to upstream nameservers and validates responses
For DHCP systems running local caching name service, upstream nameservers specified in unbound configuration may become outdated
To automatically update forwarder settings in unbound configuration file to point to new DNS servers, use dnssec-trigger
- Using dnssec-trigger with unbound helps secure DNS name resolution on roaming client machines

References

Man pages: unbound(8), unbound-checkconf(8), unbound.conf(5), unbound-control(8), and dnssec-trigger(8)

Transcript:

In addition to providing caching name service for a local subnet, unbound can also be useful as a local caching nameserver to provide secure DNS name resolution for local use on an individual system. For a local caching nameserver setup, the nameserver entry in /etc/resolv.conf will be configured to point to localhost where unbound is listening. The unbound configuration will forward DNS requests to upstream nameservers and validate their responses.

For DHCP systems running local caching name service, the upstream nameservers specified in unbound's configuration may become outdated if DNS servers provided by DHCP change. The dnssec-trigger tool supplied by the package of the same name can be leveraged to automatically update forwarder settings in unbound's configuration file to point to the new DNS servers. The use of the dnssec-trigger tool in conjunction with unbound is mostly useful for secure DNS name resolution on roaming client machines.

Troubleshooting DNS

DNS name resolution depends on proper configuration/operation of DNS on:
- System
- Resolving nameserver
- Authoritative nameservers used to resolve DNS requests
Recursive name resolution involves interactions with many different authoritative nameservers
- Creates many possible points for failure

Troubleshooting DNS

Caching nameservers reduce DNS workloads and improve DNS performance
- Caching function adds another point of failure
- Creates scenarios where DNS responses received by clients are inaccurate because data is no longer current
Must be able to quickly resolve DNS issues to minimize service interruptions
Key to DNS troubleshooting is pinpointing which point in client-server interactions is responsible for unexpected behavior
Requires proper tools and understanding diagnostic data they provide
Domain Internet Groper (dig) is good tool for investigating DNS issues

Transcript:

The use of caching nameservers significantly reduces DNS workloads and improves DNS performance. However, the caching function adds another point of failure by creating scenarios where it is possible for DNS responses received by clients to be inaccurate due to the data no longer being current.

Due to the critical role that DNS plays in the functioning of network services, it is important that Linux administrators be able to quickly resolve DNS issues when they occur, in order to minimize service interruptions. The key to accurate and efficient DNS troubleshooting is being able to pinpoint which of the multiple points in the myriad of behind-the-scenes client-server interactions is responsible for the unexpected behavior observed. This requires the use of proper tools and a clear understanding of the diagnostic data they provide. Domain Internet Groper (dig) is a good tool for investigating DNS issues due to its verbose diagnostic output.

Name Resolution Methods

DNS service often bears blame for unexpected name resolution results
Name resolution on networked hosts can occur via other methods:
- Local hosts files
- Windows Internet Name Service (WINS)
- Etc.
By default, Linux attempts name resolution with /etc/hosts
- Resolves based on order in /etc/nsswitch.conf
Do not assume issue resides with DNS
- First identify which name resolution mechanism is in play

Name Resolution Methods

To perform name resolution, use getent from glibc-common package or gethostip from syslinux package
- Mirrors process used by applications in following order of host name resolution as dictated by /etc/nsswitch.conf
  [student@desktop1 ~]$ getent hosts example.com 192.168.0.254 example.com
  [student@desktop1 ~]$ gethostip example.com example.com 192.168.0.254 AC19FEFE
If getent or gethostip result differs from dig result, something besides DNS is responsible
Consult /etc/nsswitch.conf to determine other name resolution mechanisms employed before DNS

Client/Server Network Connectivity

For proper DNS name resolution, system must conduct client/server interactions with resolving nameserver or authoritative nameservers
Common DNS issues at this layer result from resolver and firewall misconfigurations

When using dig to troubleshoot issue:

If no response received from DNS server, cause lies with client/server network connectivity to DNS server

[student@desktop1 ~]$ dig A example.com
; <<>> DiG 9.9.4-RedHat-9.9.4-14.el7 <<>> A example.com
;; global options: +cmd
;; connection timed out; no servers could be reached

Possible cause: Inability to reach DNS server due to incorrect DNS server IP address(es) in DNS configuration
- If system acts as DNS client, check /etc/resolv.conf
- If system is configured as unbound caching nameserver, check forward-zone clause of /etc/unbound/unbound.conf

Transcript:

For DNS name resolution to work properly, a system must first be able to conduct client-server interactions with its resolving nameserver or other authoritative nameservers. Some common DNS issues that have their origin at this layer are the result of resolver and firewall misconfigurations.

When using dig to troubleshoot a DNS issue, if a response is not received from a DNS server, it is a clear indication that the cause lies with the client-server network connectivity to the DNS server.

A possible cause is the inability to reach the DNS server due to incorrect DNS server IP address(es) in a system’s DNS configuration. This could be in /etc/resolv.conf, in the case of a system acting as a DNS client, or in the forward-zone clause of /etc/unbound/unbound.conf, in the case of a system configured as an unbound caching nameserver.

Client/Server Network Connectivity

Firewall Rules

Another possible cause: Firewall rules on client or server system blocking DNS traffic on port 53
- DNS mostly uses UDP protocol
- When response data sizes exceed 512 bytes, resolvers use TCP to retry query
  - Or responses exceed 4096 bytes for DNS servers that support EDNS
- DNS configuration should allow for DNS traffic on port 53 for UDP and TCP
- Allowing port 53 traffic for UDP only results in truncation error if resolver encounters response larger than it can handle over UDP
  [student@desktop1 ~]$ dig @server1.example.com A labhost1.example.com ;; Truncated, retrying in TCP mode. ;; Connection to 192.168.0.101#53(192.168.0.101) for labhost1.example.com failed: host unreachable.

Client/Server Network Connectivity

dig

To troubleshoot whether DNS queries can succeed with TCP, use dig's tcp or vc options
- Force dig to use TCP regardless of response size
  [student@desktop1 ~]$ dig +tcp A example.com
For DNS issues at network layer, dig provides sparse output
- Often useful to also use network packet analyzer such as tcpdump to determine what is transpiring at network layer
- tcpdump lets you determine information such as:
  - Destination IP address of DNS request
  - If request packets leave client
  - If request packets reach server
  - If response packets leave server
  - If response packets reach client

Transcript:

dig's tcp or vc options are helpful for troubleshooting whether DNS queries can succeed with TCP. These options force dig to use TCP, rather than the default behavior of using UDP first and falling back to TCP only when response size necessitates it.

When dealing with DNS issues at the network layer, dig provides very sparse output and it is therefore often useful to also use a network packet analyzer, such as tcpdump, to determine what is transpiring behind the scenes at the network layer. Using tcpdump, the administrator can determine information that cannot be ascertained with dig alone, such as the destination IP address of the DNS request, if request packets leave the client, if request packets reach the server, if response packets leave the server, if response packets reach the client, etc.

DNS Response Codes

If DNS client/server communication successful, dig generates output detailing nature of response received from DNS server

status field in HEADER section reports response code generated by DNS server

[student@desktop1 ~]$ dig A example.com
; <<>> DiG 9.9.4-RedHat-9.9.4-14.el7 <<>> A example.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 30523
...

NOERROR status indicates query was resolved successfully
If server encounters problems fulfilling query, error status is displayed:

Code

Meaning

SERVFAIL

Nameserver encountered problem while processing query

NXDOMAIN

Queried name does not exist in zone

REFUSED

Nameserver refused client’s DNS request due to policy restrictions.

DNS Response Codes

SERVFAIL

Common cause of SERVFAIL is failure of DNS server to communicate with authoritative nameservers for queried name
May be due to:
- Authoritative nameservers being unavailable
- Problem at network layer interfering with client/server communication between DNS server and nameservers
  - Examples: Metwork routing issues or firewall rules at network path hop
To determine cause for SERVFAIL, determine which nameserver communication is causing failure
To see results of nameserver’s iterative queries starting with root nameservers, use dig's +trace option

Transcript:

A common cause of SERVFAIL status is the failure of the DNS server to communicate with the nameservers authoritative for the name being queried. This may be due to the authoritative nameservers being unavailable. It could also be a problem at the network layer interfering with the client-server communication between the DNS server and the authoritative nameservers, such as network routing issues or firewall rules at any hop in the network path.

To determine why a nameserver is generating a SERVFAIL status while performing recursion on behalf of a client’s query, the administrator of the nameserver will need to determine which nameserver communication is causing the failure. dig's +trace option is helpful in this scenario to see the results of nameserver’s iterative queries starting with the root nameservers.

DNS Response Codes

NXDOMAIN

NXDOMAIN indicates no records were found associated with name queried
If result is not expected result and query is directed at server not authoritative for name, server’s cache may contain negative cache for name
Can either:
- Wait for server to expire negative cache of name
- Submit request to server administrator to flush name from cache
- After name removed from cache, server queries authoritative nameserver to receive current resource records for name
NXDOMAIN may also occur when querying CNAME record containing orphaned CNAME
Name on right side of CNAME record should point to name containing A or AAAA records
If records are nonexistent or removed, canonical name in CNAME is orphaned
- Queries for owner name in CNAME record no longer resolvable and result in NXDOMAIN

Transcript:

An NXDOMAIN status indicates that no records were found associated with the name queried. If this is not the expected result and the query is directed at a server that is not authoritative for the name, then the server’s cache may contain a negative cache for the name. The user can either wait for the server to expire the negative cache of that name, or submit a request to the server administrator to flush the name from the server’s cache. Once the name is removed from cache, the server will query the authoritative nameserver to receive current resource records for the name.

The other scenario where an NXDOMAIN status may be unexpectedly encountered is when querying a CNAME record containing an orphaned CNAME. In a CNAME record, the name on the right side of the record, the canonical name, should point to a name that contains either A or AAAA records. If these associated A or AAAA records are nonexistent or are later removed, then the canonical name in the CNAME record is orphaned. When this occurs, queries for the owner name in the CNAME record will no longer be resolvable and will result in an NXDOMAIN return code.

DNS Response Codes

REFUSED

REFUSED indicates DNS server has policy restriction that keeps it from fulfilling client’s query
Policy restrictions often implemented to restrict clients that can make recursive queries and zone transfer requests
Common causes of unexpected REFUSED status:
- Clients configured to query wrong DNS servers
- DNS server misconfiguration causing valid client requests to be refused

Other Common DNS Issues

Outdated Cached Data

NOERROR signifies no errors in resolving query
- Does not guarantee no DNS issues present
DNS records in DNS response may not match expected result
- Common cause: Answer originated from outdated cached data

To if response is authoritative cached data, review dig output

Authoritative answers have aa flag:

[student@desktop1 ~]$ dig A example.com
; <<>> DiG 9.9.4-RedHat-9.9.4-14.el7 <<>> A example.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 22257
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 2

Other sign that answer is coming from cache is counting down resource record TTL value in responses of each subsequent query
- TTLs of cached data count down to expiration
- TTLs of authoritative data remain static

Transcript:

A DNS return code of NOERROR signifies that no errors were encountered in resolving a query. However, it does not guarantee that there are no DNS issues present. There are situations where the DNS records in the DNS response may not match the expected result. The most common cause for an incorrect answer is that the answer originated from cached data, which is no longer current.

In these situations, first confirm that the response is indeed nonauthoritative cached data. This can be easily determined by looking at the flags section of dig's output. DNS responses, which are authoritative answers, will be indicated by the presence of the aa flag.

DNS responses that originate from cached data are not authoritative and therefore will not have the aa flag set. The other telltale sign that an answer is coming from cache is the counting down of the resource record’s TTL value in the responses of each subsequent query. TTLs of cached data will continuously count down to expiration while TTLs of authoritative data will always remain static.

Other Common DNS Issues

Responses for Nonexistent Records

If record was removed from zone but response still received when querying for record:
- Confirm queries not being answered from cached data
- If aa flag is present in dig output, wildcard (*) record may be present in zone:
  *.example.com. IN A 192.168.0.254
Wildcard record serves as catchall for all queries of given type for nonexistent name
With wildcard record in place:
- If A record existed for server1.example.com and is removed
- Queries for name succeed and IP address in wildcard A record is provided

Other Common DNS Issues

Non-FQDN Name Errors

In zone file, host names not expressed as Fully Qualified Domain Names (FQDNs) automatically expand to FQDNs by appending zone name
To indicate that name is FQDN in zone file, name must end in . (www.example.com.)
Not doing so can lead to different issues depending on record type
- Mistake in type-specific data portion of NS records can incapacitate entire zone
- Mistake in MX records can halt email delivery for domain

Other Common DNS Issues

Looping CNAME Records

Avoid CNAME records that point to CNAME records
- Reduces DNS lookup inefficiency
- Possibly creates unresolvable CNAME loops:
  test.example.com. IN CNAME lab.example.com. lab.example.com. IN CNAME test.example.com.
CNAME record with orphaned CNAME results in NXDOMAIN status
Looping CNAME records return NOERROR

Other Common DNS Issues

Missing PTR Records

Network services use DNS to perform reverse lookups of incoming client connections
Absence of PTR records in DNS may result in different issues depending on service:
- SSHD performs reverse lookups of connecting client IPs
  - Absence of PTR records leads to delays in establishing connections
- MTAs incorporate reverse DNS lookups of connecting client IPs as defense against malicious email clients
- Many MTAs configured to reject client connections for IPs
  - Cannot be resolved with PTR query in DNS
Understand requirements network services have for forward and reverse DNS lookups

Other Common DNS Issues

Round-Robin DNS

Round-robin DNS: Name has multiple A or AAAA records in DNS
- Simple, low-cost, load-balancing mechanism to distribute network resource loads across multiple hosts
When DNS client queries name containing multiple A/AAAA records, all records return as a set
- Record order in set permutates for each query
- Clients normally use first address in set
- Variation in record order results in distribution of network service across IP addresses
Can inadvertently create round-robin DNS configuration
- If request to change IP address of A record is mistakenly implemented as resource record addition instead of resource record modification
If network resources on old IP address are retired, load distribution effect of round-robin DNS results in service failures for half of clients

References

Man pages: dig(1), getent(1), and gethostip(1)

Transcript:

A name can have multiple A or AAAA records in DNS. This is known as round-robin DNS, and is often used as a simple, low-cost, load-balancing mechanism to distribute network resource loads across multiple hosts. When a DNS client queries for a name that contains multiple A or AAAA records, all records are returned as a set. However, the order that the records are returned in the set permutates for each query. Since clients normally make use of the first address in the set, the variation in the order of the records in each response effectively results in a distribution of network service requests across the multiple IP addresses in these round-robin DNS records.

While round-robin DNS is a valid technical configuration, there are times when this configuration is inadvertently created. When a request to change the IP address of an A record is mistakenly implemented as a resource record addition rather than a resource record modification, then round-robin DNS is created. If the network resources on the old IP address is retired, the load distribution effect of the round-robin DNS will result in service failures for approximately half of the clients.

Summary

Domain Name System
Anatomy of DNS Lookups
DNS Resource Records
Caching Nameservers and DNSSEC
Configuring unbound
Dumping and Loading unbound Cache
Flushing unbound Cache
Updating Local Caching unbound Configuration With dnssec-trigger
Troubleshooting DNS
Name Resolution Methods
Client/Server Network Connectivity
DNS Response Codes
Other Common DNS Issues

Module Completion

Nice job!

Click the button below to complete this module of the course:

Red Hat Enterprise Linux Implementation

DNS Management for Servers

Module Topics

Domain Name System

Domain Name System

Domain Name System

Anatomy of DNS Lookups

Anatomy of DNS Lookups

Anatomy of DNS Lookups

DNS Resource Records

DNS Resource Records

DNS Resource Records

DNS Resource Records

DNS Resource Records

DNS Resource Records

DNS Resource Records

DNS Resource Records

DNS Resource Records

DNS Resource Records

DNS Resource Records

DNS Resource Records

DNS Resource Records

DNS Resource Records

DNS Resource Records

DNS Resource Records

Caching Nameservers and DNSSEC

Caching Nameservers and DNSSEC

Configuring unbound

Configuring unbound

Configuring unbound

Configuring unbound

Dumping and Loading unbound Cache

Flushing unbound Cache

Updating Local Caching unbound Configuration With dnssec-trigger

Troubleshooting DNS

Troubleshooting DNS

Name Resolution Methods

Name Resolution Methods

Client/Server Network Connectivity

Client/Server Network Connectivity

Client/Server Network Connectivity

DNS Response Codes

DNS Response Codes

DNS Response Codes

DNS Response Codes

Other Common DNS Issues

Other Common DNS Issues

Other Common DNS Issues

Other Common DNS Issues

Other Common DNS Issues

Other Common DNS Issues

Summary

Module Completion

Configuring `unbound`

Configuring `unbound`

Configuring `unbound`

Configuring `unbound`

Dumping and Loading `unbound` Cache

Flushing `unbound` Cache

Updating Local Caching `unbound` Configuration With `dnssec-trigger`